Switch to the LLVM SPIR-V back-end. by maleadt · Pull Request #491 · JuliaGPU/oneAPI.jl

maleadt · 2025-02-13T09:00:26Z

PoC, requires quite some work

exp intrinsic is broken

PHI node operands are not the same type as the result!
  %common.ret.op = phi [2 x float] [ %177, %L5 ], [ %192, %L28 ], [ %201, %L41 ], [ %213, %L53 ], [ %4, %L25 ], [ %8, %L19 ]
Function return type does not match operand type of return inst!
  ret [2 x float] %common.ret.op
 i32in function julia_exp_27849

atomics probably have to be ported to native LLVM atomics
the lack of OpCapability Linkage results in ZE_RESULT_ERROR_INVALID_MODULE_UNLINKED (I think)

maleadt · 2025-02-13T11:37:06Z

the lack of OpCapability Linkage results in ZE_RESULT_ERROR_INVALID_MODULE_UNLINKED (I think)

The actual issue is our use of OpenCL intrinsic functions to access things like the work dimension. We'll have to implement get_work_dim as an access to @__spirv_BuiltInWorkDim = external addrspace(1) global i32 instead of call to _Z12get_work_dim.

maleadt · 2025-07-10T07:30:18Z

Rebased. Things work, however, this triggers multiple crashes in the Intel compiler stack. Anybody who would want to look into this would need to (first upgrade the compiler stack, but if things persist then) reduce the issues to minimal issues and file them upstream.

michel2323 · 2025-08-22T15:55:47Z

@maleadt We released a new oneAPI.jl version with NEO 25 and then rebased this branch. Now I get error: undefined reference to '_Z17__spirv_IAddCarryll'. Since this looks like a missing intrinsic I tried to bump SPIRVInstrinsics. However, in later versions this requires a method_table in oneAPI. I've tried to add that too, but I can't get the kernels to pick up anything and they return zeros. Mainly, I was then fiddling with @device_override, but I just got lost. Can you help a long here a bit?

michel2323 · 2025-08-29T14:41:38Z

See: JuliaGPU/OpenCL.jl#316

michel2323 · 2025-09-03T17:53:40Z

@maleadt I've rebased this again cleanly onto master.

michel2323 · 2025-10-15T15:10:28Z

Rebase and now we have an assortment of failures:

Julia 1.10: Intel runtime craps out in L0
Julia 1.11: The _Z17__spirv_IAddCarryll error
Julia 1.12: Silently produces wrong results

I'm inclined to work on 1.12 and make this work some time in a future version.

codecov · 2026-06-17T10:15:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.79%. Comparing base (8a3e4b3) to head (c610b8d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #491      +/-   ##
==========================================
+ Coverage   79.74%   80.79%   +1.04%     
==========================================
  Files          48       48              
  Lines        3234     3233       -1     
==========================================
+ Hits         2579     2612      +33     
+ Misses        655      621      -34

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Aurora ships the Intel "LTS" GPU software stack rather than the rolling release that upstream oneAPI.jl targets. Pin the whole toolchain to it: - JLLs: NEO_LTS_jll =25.18.33578, oneAPI_Level_Zero_Loader_LTS_jll =1.24, oneAPI_Level_Zero_Headers_LTS_jll, libigc_LTS_jll, and the oneMKL 2025.3.1 support library. lib/level-zero/oneL0.jl aliases the *_LTS_jll packages back to their canonical names so the rest of the code is unchanged. - Keep the SPIR-V Translator (SPIRV_LLVM_Translator_jll 21, SPIRVIntrinsics 0.5). The LTS NEO/IGC runtime does not accept the output of upstream's LLVM SPIR-V back-end (#491), so revert the back-end switch in src/compiler/compilation.jl, src/oneAPI.jl and src/utils.jl and restore the explicit SPIR-V extension list. - Regenerate the oneMKL C bindings against 2025.3.1 (deps/src/onemkl.*, lib/support/liboneapi_support.jl, deps/generate_interfaces.jl) and fix the sparse CSR argument types (ZePtr/onemklIndex) in the regenerated bindings and res/support.toml. Stay on oneMKL 2025.3.1, declining the upstream oneAPI 2026.0.0 migration (#582). - Prepend NEO's libze_intel_gpu directory to LD_LIBRARY_PATH in __init__ so libsycl's bundled ze_lib finds the driver by path when no system NEO is installed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The LTS NEO/IGC runtime does not accept the output of the LLVM SPIR-V back-end (#491) and needs SPIR-V extensions declared explicitly, which the back-end handles itself. Select the codegen path from oneL0.LTS[]: :khronos (translator) on the LTS stack, :llvm on the rolling stack. GPUCompiler resolves the tool from the target's `backend` field through a LazyModule that looks the JLL up in Base.loaded_modules, so both JLLs are listed as deps and loaded, and the choice is made at compile time. Keep SPIRV_LLVM_Backend_jll: with it neither a dep nor loaded, the first @oneapi launch on the rolling stack fails in mcgen and no kernel compiles at all. Preserve the rolling-stack behavior exactly: supports_bfloat16 stays _device_supports_bfloat16(), SPV_KHR_bfloat16 is declared when the runtime advertises it, and finish_ir! otherwise lowers bfloat->i16. Force bf16 off on the LTS branch, whose SPIR-V stack cannot codegen native bfloat in generic kernels (clamp! fails with InvalidIRError, and declaring SPV_KHR_bfloat16 crashes the LTS runtime). The precompile workload mirrors the same choice, resolving ONEAPI_LTS directly from the environment since oneL0.__init__ has not run yet.

Aurora and other large deployments run Intel's long-term-servicing branch of the Compute Runtime rather than the rolling release oneAPI.jl targets. That branch predates a number of driver and IGC fixes, several of which corrupt results silently rather than raise an error. Add the workarounds behind a single opt-in switch, oneL0.LTS[], resolved from ONEAPI_LTS at load time and defaulting to off, so the rolling stack behaves exactly as before. Gated on that switch: * SPIR-V codegen goes through the Khronos translator instead of LLVM's SPIR-V back-end, whose output the LTS NEO/IGC runtime does not accept (#491). Both JLLs stay dependencies; GPUCompiler resolves the tool from the target's `backend` field, so the choice is made at compile time. * BFloat16 is forced off. The LTS SPIR-V stack cannot codegen native bfloat in generic kernels -- a kernel that merely keeps a bfloat value fails with InvalidIRError, and declaring SPV_KHR_bfloat16 crashes the runtime. _device_supports_bfloat16() reports hardware capability and does not capture this, so the test suite and examples/bfloat16.jl gate on the switch instead. * Reductions avoid the strided-read miscompile that silently breaks `sum(transpose(x))`, `a == transpose(b)` and `ishermitian`. Non-dense inputs are materialized before reducing, and reductions that keep the contiguous leading dimension use a coalesced one-work-item-per-slice kernel. Reductions that also reduce dim 1 keep a contiguous innermost axis and are left alone; an Int32 regression test covers that boundary. * Buffers are freed only after draining the queues that may still reference them. LTS NEO advertises ZE_extension_memory_free_policies but ignores BLOCKING_FREE, so a GC-driven free of in-flight work faults and bans the context, surfacing later as a ZE_RESULT_ERROR_UNKNOWN at an unrelated call. A per-(context, device) registry tracks every queue -- including the replacement KA.priority! installs -- and the queue finalizer drains before destroying, with a bounded wait so a task that dies mid-submission cannot hang finalization. Independently of that switch, ONEAPI_SYNC_EACH_SUBMISSION=1 synchronizes after every command-list submission, working around a dropped-tail corruption seen when several processes oversubscribe a single tile. It costs roughly 3x throughput and is off by default, with getter, setter and scoped forms available at runtime. Also, not specific to the LTS stack: dlopen the NEO driver by full path during __init__ so libsycl's bundled Level Zero loader finds it in-process. Setting LD_LIBRARY_PATH there never could -- glibc captures it once at process startup -- so it only ever served spawned worker processes. The self-hosted Aurora runner is configured to exercise the LTS path, with buildkite continuing to cover the rolling stack, and docs/src/lts.md documents both switches, what they change, and what they cost.

maleadt mentioned this pull request Feb 13, 2025

Switch to the LLVM SPIR-V back-end. JuliaGPU/OpenCL.jl#285

Merged

maleadt force-pushed the tb/llvm_spirv_backend branch from 1d1f809 to 805799c Compare July 10, 2025 07:29

maleadt mentioned this pull request Jul 11, 2025

Use stacked method tables #506

Closed

amontoison force-pushed the tb/llvm_spirv_backend branch from 805799c to 8beebc5 Compare August 21, 2025 18:48

michel2323 force-pushed the tb/llvm_spirv_backend branch from 5c96f50 to e506217 Compare August 22, 2025 15:54

michel2323 force-pushed the tb/llvm_spirv_backend branch from e506217 to 36260b1 Compare August 29, 2025 13:37

michel2323 force-pushed the tb/llvm_spirv_backend branch 2 times, most recently from 62481c2 to 87b2e88 Compare September 3, 2025 17:52

michel2323 force-pushed the tb/llvm_spirv_backend branch from 87b2e88 to aa41a29 Compare September 10, 2025 13:17

michel2323 mentioned this pull request Oct 15, 2025

Compatibility with SPIRVIntrinsics 0.5 #540

Closed

michel2323 force-pushed the tb/llvm_spirv_backend branch from aa41a29 to 9dd1f35 Compare October 15, 2025 14:01

michel2323 mentioned this pull request Oct 28, 2025

CompatHelper: bump compat for SPIRVIntrinsics to 0.3, (keep existing compat) #505

Closed

Switch to the LLVM SPIR-V back-end.

c610b8d

maleadt force-pushed the tb/llvm_spirv_backend branch from 9dd1f35 to c610b8d Compare June 17, 2026 09:26

maleadt marked this pull request as ready for review June 17, 2026 10:48

maleadt merged commit 8a688fd into main Jun 17, 2026
5 of 6 checks passed

maleadt deleted the tb/llvm_spirv_backend branch June 17, 2026 10:48

Navigation Menu

Uh oh!

Switch to the LLVM SPIR-V back-end. - #491

Switch to the LLVM SPIR-V back-end.#491
maleadt merged 1 commit into
mainfrom
tb/llvm_spirv_backend

maleadt commented Feb 13, 2025

Uh oh!

maleadt commented Feb 13, 2025

Uh oh!

maleadt commented Jul 10, 2025

Uh oh!

michel2323 commented Aug 22, 2025

Uh oh!

michel2323 commented Aug 29, 2025 •

edited

Loading

Uh oh!

michel2323 commented Sep 3, 2025

Uh oh!

michel2323 commented Oct 15, 2025

Uh oh!

codecov Bot commented Jun 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

maleadt commented Feb 13, 2025

Uh oh!

maleadt commented Feb 13, 2025

Uh oh!

maleadt commented Jul 10, 2025

Uh oh!

michel2323 commented Aug 22, 2025

Uh oh!

michel2323 commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michel2323 commented Sep 3, 2025

Uh oh!

michel2323 commented Oct 15, 2025

Uh oh!

codecov Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

michel2323 commented Aug 29, 2025 •

edited

Loading

codecov Bot commented Jun 17, 2026 •

edited

Loading